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I.  INTRODUCTION 


This  final  technical  report  prepared  under  Contract  No. 
N00014-79-C-0650  summarizes  a  research  investigation  conducted  by 

Desmatics,  Inc.  under  sponsorship  of  the  Office  of  Naval  Research. 
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y  The  object  of  tilt  Desmalivo1  research  program  has  been  to  evaluate  and 
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compare  the  performance  of  factor  screening  procedures  for  use  in 

** 

computer  simulation  experiments.  <— - - -  \3 

The  following  sections  Jf'This  report  briefly  discuss^the  prob¬ 
lem  background,  summarize ^the  research  goals,  and  provide^a  reference 
list  of  all  technical  reports,  journal  publications,  and  presentations 
prepared  under  this  contract. 


PROBLEM  BACKGROUND 


Computer  simulation  is  often  used  to  study  real-world  systems 
that  are  too  complex  to  be  modeled  and  analyzed  entirely  by  mathe¬ 
matical  methods.  Unfortunately,  simulation  models  (used  to  study 
large,  complex  systems)  tend  to  be  extremely  large  and  complex  them¬ 
selves,  Further,  corresponding  computer  programs  (codes)  used  to 
execute  these  models  usually  contain  large  numbers  of  factors  (i.e,, 
inputs).  In  addition,  these  programs  can  be  extremely  costly  and 
time-consuming  to  run.  Users  of  such  simulations  are  often  unsure 
of  how  to  obtain  an  effective  analysis  of  the  system  model  without 
having  to  perform  an  excessive  number  of  costly  and  time-consuming 
simulation  runs.  Resource  considerations  dictate  that  methods  of 
lessening  these  cost  and  time  burdens  be  used  in  order  for  any  fruit¬ 
ful  simulation  experimentation  to  take  place. 

^  Factor  screening  methods  are  statistical  methods  that  attempt 
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to  identify,  efficiently  and  economically,  the  set  of  most  important^ 
factors.  Once  the  most  important  factors  have  been  identified,  further 
simulation  (or  real-world)  experimentation  can  focus  on  these  parti¬ 
cular  factors.  This  eliminates  experimentation  with  relatively  unimpor¬ 
tant  factors,  which  can  needlessly  consume  resources.  - 

Many  resource-efficient  designs  have  been  proposed  for  conducting 
screening  experiments.  Latin  squares  and  fractional  factorial  designs 
are  among  the  more  traditional  designs  developed  and  proposed  for  use 
in  screening  experiments.  These  designs,  however,  are  generally  not 
applicable  in  the  study  of  large-scale  simulations  because  they  require 
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more  computer  runs  than  are  normally  reasonable  or  affordable.  In 
large-scale  simulation  studies,  the  number  of  computer  runs  is  usually 
severely  limited;  in  fact,  the  number  of  factors,  K,  usually  exceeds 
the  number  of  screening  runs,  R,  that  can  be  made.  In  statistical 
design  terminology,  experiments  in  which  R  K  are  called  "super¬ 
saturated."  This  situation  has  received  relatively  little  attention 
in  the  statistical  and  simulation  literature. 


III.  RESEARCH  SUMMARY 


The  primary  goals  of  the  Desmatics  research  were  to  (I)  review 
the  major  types  of  supersaturated  screening  designs,  (2)  select  prom¬ 
ising  procedures  for  detailed  evaluation,  and  (3)  compare  the  perfor¬ 
mance  of  the  selected  strategies.  Three  strategies  were  subsequently 
selected  for  detailed  study.  Two  of  these  were  based  on  random  bal¬ 
ance  sampling  and  the  other  on  two-stage  group  screening.  The  Desmatics 
research  findings  are  described  in  a  number  of  technical  reports  and 
journal  articles.  For  the  simulation  user  contemplating  a  screening 
experiment,  these  documents  offer  Insight  into  the  various  techniques 
and  provide  quantitative  information  on  the  trade-offs  involved  in 
selecting  an  appropriate  screening  procedure. 

The  following  pages  contain  a  complete  listing  of  the  technical 
reports  prepared  under  this  contract.  For  each  report,  the  following 
information  is  provided:  Desmatics  Technical  Report  Number,  Report 
Date,  DTIC  Number,  Title,  Author(s),  and  Abstract.  A  listing  of 
journal  publications  and  technical  presentations  is  given  in  Section 


Desmatics  Report  Number:  113-1  Date:  March  1980 

DT1C  Number:  AD-A082182 

Title:  PERFORMANCE  EVALUATION  OF  A  SPECIFIC  FACTOR  SCREENING 
TECHNIQUE 

Author:  Dennis  E.  Smith 

Abstract:  In  many  experimental  situations  (particularly  in  computer 

simulation  studies)  a  large  number  of  potentially  important 
factors  exist.  Because  of  time  and  budget  limitations,  it 
is  imperative  to  screen  these  factors  in  order  to  identify 
a  subset  which  should  be  subjected  to  more  detailed  exami¬ 
nation.  This  paper  evaluates  the  performance  of  a  factor- 
screening  technique  which  has  been  proposed  for  use  when  it 
is  known  that  there  is  at  most  one  active  factor  (i.e.,  a 
factor  which  has  an  effect  on  the  response  of  interest). 
Performance  evaluation  reveals  that  the  existence  of  even 
a  relatively  small  amount  of  random  error  renders  essentially 
useless  a  procedure  which  performs  well  in  the  deterministic 
case. 


Desmatics  Report  Number:  113-2  Date:  July  1980 

DTIC  Number:  AD-A086449 

Title:  FACTOR  SCREENING  IN  COMPUTER  SIMULATIONS:  CONSIDERATIONS 
IN  PERFORMANCE  EVALUATION 

Authors:  Dennis  E.  Smith  and  Carl  A.  Mauro 

Abstract:  Although  computer  simulation  is  a  major  area  in  which  fac¬ 
tor  screening  situations  are  frequently  encountered,  ade¬ 
quate  methodology  has  not  been  developed  to  resolve  the 
factor  screening  problem.  Within  the  constraint  of  a 
limited  number  of  computer  runs,  a  decision  must  be  made 
about  the  selection  of  a  screening  strategy  that  will  per¬ 
form  efficiently  and  effectively.  This  report  reviews  the 
major  classes  of  screening  designs  that  have  been  suggested 
and  recommends  two  strategies  for  further  research.  In 
addition,  a  fundamental  screening  model  is  defined,  and  a 
performance  measure  for  evaluating  screening  strategies 
is  developed. 


Desmatics  Report  Number:  113-3 


Date:  October  1980 


DTIC  Number:  AD-A091553 

Title:  AN  EXAMINATION  OF  THE  PERFORMANCE  OF  TWO-STAGE  GROUP 
SCREENING  FOR  USE  IN  COMPUTER  SIMULATION 

Authors:  Carl  A.  Mauro  and  Dennis  E.  Smith 

Abstract:  When  only  a  limited  number  of  simulation  runs  are  avail¬ 
able*  it  is  usually  Impossible  to  thoroughly  investigate 
all  of  the  factors  under  consideration.  Often,  though, 
it  is  anticipated  that  only  a  small  subset  of  the  original 
factors  is  important.  Accordingly,  it  is  desired  to  screen 
the  factors  in  order  to  help  identify  those  that  do  exert 
an  appreciable  effect  on  the  simulation  response.  Two- 
stage  group  screening  is  one  possible  strategy  that  may  be 
used.  However,  a  basic  assumption  of  this  strategy  is 
that  the  directions  of  possible  effects  are  known,  a  priori. 
This  report  examines,  in  the  case  of  zero  error  variance 
(i.e.,  when  the  simulation  response  is  observed  without 
random  error),  the  performance  of  two-stage  group  screening 
when  the  assumption  of  known  directions  is  relaxed. 


Desmatics  Report  Number:  113-4  Date:  May  1981 

DTIC  Number:  AD-A099703 

Title:  THE  PROBLEM  OF  EXPERIMENTAL  DESIGN  IN  SIMULATION 
Authors:  Dennis  E.  Smith  and  Carl  A.  Mauro 

Abstract:  In  many  cases,  a  simulation  study  may  be  viewed  as  an 

experimental  situation  in  which  a  number  of  factors  (in¬ 
dependent  variables)  are  to  be  investigated.  However, 
standard  experimental  design  techniques  often  require 
more  simulation  runs  than  are  available  to  the  simulation 
user.  In  general,  the  primary  problem  of  experimental 
design  in  simulation  can  be  succinctly  summarized  as  too 
many  factors  and  too  few  runs.  A  discussion  of  this  problem 
is  presented  in  this  report.  Three  possible  two-stage 
strategies  for  attacking  the  problem  are  considered,  and 
performance  measures  with  which  to  judge  the  strategies 
are  described. 


Desmatics  Report  Number:  113-5  Date:  August  1^81 

DTIC  Number:  AD-A102763 

Title:  FACTOR  SCREENING  IN  SIMULATION:  PERFORMANCE  OF  A  TWO-STAGE 
RANDOM  BALANCE/PLACKETT-BURMAN  PROCEDURE 

Authors:  Carl  A.  Mauro  and  Dennis  E.  Smith 

Abstract:  In  simulation  models  Involving  many  factors  (input  variables) , 
usually  relatively  few  are  important.  Often  it  is  desired 
to  "screen"  out  these  important  factors  using  a  limited 
number  of  computer  runs.  This  paper  examines  the  performance 
of  a  two-stage  screening  procedure  based  on  random  balance 
and  Plackett-Burman  designs. 


Desmatics  Report  Number:  113-6  Date:  July  1982 

DTIC  Number:  AD-A1 17529 

Title:  FACTOR  SCREENING  IN  SIMULATION:  EVALUATION  OF  A  RANDOM 
BALANCE/PLACKETT-BURMAN  STRATEGY 

Authors:  Carl  A.  Mauro  and  Dennis  E.  Smith 

Abstract:  Large,  complex  simulation  models  usually  must  be  studied 
by  performing  a  number  of  simulation  runs.  It  is  often 
to  our  advantage  to  invest  a  relatively  small  number  of 
runs  in  a  preliminary  experiment  aimed  at  screening  out 
the  most  important  factors.  The  major  experimental  effort 
and  expense  can  then  be  focused  on  these  key  factors.  In 
this  report  we  evaluate  the  performance  of  a  two-stage 
screening  strategy  that  is  based  on  a  combination  of  random 
balance  and  Plackett-Burman  designs. 


Desmatics  Report  Number:  113-7  Date:  November  \ZB2 

DT1C  Number:  AD-A121874 

Title:  FACTOR  SCREENING  IN  SIMULATION:  EVALUATION  OF  TWO  STRATEGIES 
BASED  ON  RANDOM  BALANCE  SAMPLING 

Authors:  Carl  A.  Mauro  and  Dennis  E.  Smith 

Abstract:  In  the  study  of  large,  complex  computer  simulation  models 
the  user  is  often  overwhelmed  by  the  vast  number  of  input 
variables.  Moreover,  he  or  she  is  usually  confused  about 
how  to  make  an  effective  analysis  of  the  model  without  per¬ 
forming  an  excessive  number  of  runs,  which  tend  to  be  costly 
and  time  consuming.  Factor  screening  methods,  which  attempt 
to  identify  the  more  important  variables,  can  be  extremely 
useful  in  the  study  of  such  models.  This  paper  presents  and 
evaluates  two  screening  strategies  based  upon  random  balance 
sampling.  Both  strategies  are  applicable  when  there  are 
more  variables  to  be  screened  than  there  are  available 
screening  runs.  The  results  provide  guidance  in  using 
these  strategies  in  particular  screening  applications. 


Desmatics  Report  Number:  113-8  Date:  May  1983 

DTIC  Number:  AD-A129109 

Title:  ON  THE  USE  OF  STAGEWISE  REGRESSION  IN  RANDOM  BALANCE 
SCREENING  EXPERIMENTS 

Author:  Carl  A.  Mauro 

Abstract:  In  this  report  we  investigate  the  use  of  stagewise  regression 
as  a  method  of  analysis  for  random  balance  screening  experi¬ 
ments.  We  also  discuss  how  this  method  compares  with  an 
individual  F-test  analysis  procedure. 


Desmatics  Report  Number:  113-9  Date:  August  1983 

DTIC  Number:  AD-A131970 

Title:  ON  THE  PERFORMANCE  OF  TWO-STAGE  GROUP  SCREENING  EXPERIMENTS 
Author:  Carl  A.  Mauro 

Abstract:  Many  experimental  situations  require  consideration  of  a 

large  number  of  factors.  For  example,  computer  simulation 
studies  are  frequently  characterized  by  the  inclusion  of 
an  exceptionally  large  number  of  input  variables.  Because 
of  resource  limitations,  there  is  often  a  need  in  such 
situations  for  an  efficient  method  of  factor  screening. 

One  possible  screening  strategy  is  two-stage  group  screening. 
This  report  attempts  to  examine  the  performance  character¬ 
istics  of  two-stage  group  screening  experiments  as  a  function 
of  a  constant  signal-to-noise  ratio  for  all  important  factors 
The  effect  of  group-factor  cancellation  on  performance  is 
also  investigated. 


Desmatics  Report  Number:  113-10  Date:  September  1983 

DTIC  Number:  AD-A133169 

Title:  TWO-STAGE  GROUP  SCREENING  SEARCH  ROUTINE:  PROGRAM  DESCRIPTION 
AND  USER'S  GUIDE 

Authors:  Carl  A.  Mauro  and  Kevin  C.  Burns 

Abstract:  A  computer-aided  search  program  has  been  developed  to 
help  select  a  satisfactory  two-stage  group  screening 
strategy.  The  program  is  easy  to  use  and  provides  prac¬ 
tical  insight  into  the  use  and  selection  of  a  two-stage 
group  screening  strategy.  This  report  describes  the  pro¬ 
gram  and  serves  as  a  user's  guide.  A  listing  of  the  pro¬ 
gram  is  included. 


Desmatics  Report  Number:  113-11  Date:  September  1983 

DTIC  Number:  AD-A133287 

Title:  A  NOTE  ON  THE  ANALYSIS  OF  RANDOM  BALANCE  DESIGNS 
Author:  Carl  A.  Mauro 

Abstract:  In  this  report  the  use  of  separate  F-tests  as  a  method 

of  analysis  foi  Random  Balance  (RB)  designs  is  discussed. 
The  validity  of  this  test  procedure  is  shown  by  relating 
the  RB  model  to  the  exchangeable  linear  model.  For  this 
latter  model ,  the  usual  F-tests  are  known  to  be  valid 
for  certain  hypotheses. 


Desmatics  Report  Number:  113-12  Date:  December  1983 

DTIC  Number:  AD-A1 36487 

Title:  A  COMPARISON  OF  RANDOM  BALANCE  AND  TWO-STAGE  GROUP  SCREENING 
DESIGNS  -  PART  I 

Author:  Carl  A.  Mauro 

Abstract:  This  report  is  the  first  of  two  reports  which  compare  the 

efficacy  and  relative  merits  of  a  two-stage  group  screening 
strategy  versus  a  random  balance  screening  strategy.  These 
techniques  for  factor  screening  have  potential  application 
in  the  study  of  large-scale  simulation  models. 


Desmatics  Report  Number:  113-13  Date:  January  1984 

DTIC  Number:  AD-A137229 

Title:  DESIGN  OF  EXPERIMENTS  IN  SIMULATION 
Authors:  Dennis  E.  Smith  and  Carl  A.  Mauro 

Abstract:  In  many  cases  a  simulation  study  may  be  viewed  as  an 
experimental  situation  in  which  a  number  of  factors 
(independent  variables)  are  to  be  investigated.  There 
are*  however,  a  number  of  special  characteristics  that 
distinguish  simulation  experiments  from  statistical 
experiments  in  general.  In  this  paper  we  discuss  four 
topics  which  we  feel  are  of  particular  interest  and  rele¬ 
vance  in  the  simulation  context.  These  are:  (1) 
Identification  of  the  important  factors  (i.e.,  input 
variables);  (2)  Investigation  of  the  statistical  re¬ 
lationship  between  the  output  and  input  variables; 

(3)  Determination  of  the  combination  of  factor  levels 
for  which  the  response  (i.e.,  output  variable)  is  opti¬ 
mized;  and  (4)  The  use  of  variance  reduction  techniques. 


Desmatics  Report  Number:  113-14  Date:  February  1984 

DTIC  Number:  AD-A137994 

Title:  A  COMPARISON  OF  RANDOM  BALANCE  AND  TWO-STAGE  GROUP  SCREENING 
DESIGNS  -  PART  II 

Authors:  Carl  A.  Mauro  and  Kevin  C.  Burns 

Abstract:  In  this  paper  we  focus  on  the  problem  of  supersaturated 
(fewer  runs  than  factors)  screening  experiments.  We 
consider  two  major  types  of  designs  which  have  been  pro¬ 
posed  in  this  situation,  namely,  random  balance  and  two- 
stage  group  screening.  We  discuss  the  relative  merits 
and  demerits  of  each  strategy.  In  addition,  we  compare 
the  performance  of  these  strategies  by  means  of  a  case 
study  in  which  K-100  factors  are  screened  in  N*20,42,62, 
and  84  runs. 


IV.  JOURNAL  ARTICLES  AND  TECHNICAL  PRESENTATIONS 


In  addition  to  the  technical  reports  listed  in  Section  III, 
significant  research  findings  have  been  documented  in  several  journal 
articles.  Research  accomplishments  have  also  been  presented  at 
scientific  and  technical  meetings.  A  complete  listing  of  journal 
publications  and  technical  presentations  prepared  under  this  contract 
is  given  below: 


A.  JOURNAL  PUBLICATIONS 


FACTOR  SCREENING  IN  COMPUTER  SIMULATION  by  Dennis  E.  Smith 
and  Carl  A.  Mauro,  Simulation,  Vol.  38,  pp.  49-54,  Feb.  1982. 

THE  PERFORMANCE  OF  TWO-STAGE  GROUP  SCREENING  IN  FACTOR  SCREENING 
EXPERIMENTS  by  Carl  A.  Mauro  and  Dennis  E.  Smith,  Technometrics, 
Vol.  24,  pp.  325-330,  Nov.  1982. 

FACTOR  SCREENING  IN  SIMULATION:  EVALUATION  OF  TWO  STRATEGIES 
BASED  ON  RANDOM  BALANCE  SAMPLING  by  Carl  A.  Mauro  and  Dennis 
E.  Smith,  Management  Science,  Vol.  30,  pp.  209-221,  Feb.  1984. 

ON  THE  PERFORMANCE  OF  TWO-STAGE  GROUP  SCREENING  EXPERIMENTS 
by  Carl  A.  Mauro,  Technometrics,  Vol.  26,  to  appear  August  1984. 

DESIGN  OF  EXPERIMENTS  IN  SIMULATION  by  Dennis  E.  Smith  and 
Carl  A.  Mauro,  American  Journal  of  Mathematical  and  Management 
Sciences,  to  appear  as  invited  paper  for  special  issue  on 
simulation. 

EXAMINATION  OF  A  PARSIMONIOUS  FACTOR  SCREENING  TECHNIQUE: 
SCREENING  K-2N-1  -1  FACTORS  IN  N  RUNS  by  Dennis  E.  Smith, 
Comnunications  In  Statistics,  Series  B,  Vol.  13,  No.  3,  1984. 


A  COMPARISON  OF  RANDOM  BALANCE  AND  TWO-STAGE  GROUP  SCREENING 
DESIGNS:  A  CASE  STUDY  by  Carl  A.  Mauro  and  Kevin  C.  Burns, 
submitted  for  publication  in  Communications  in  Statistics. 


TECHNICAL  PRESENTATIONS 


FACTOR  SCREENING  APPROACHES  IN  SIMULATION,  TIMS/ORSA  Joint 
National  Meeting,  Washington,  DC,  May  1980. 

PERFORMANCE  EVALUATION  OF  A  SPECIFIC  FACTOR  SCREENING  TECHNIQUE, 
TIMS/ORSA  Joint  National  Meeting,  Colorado  Springs,  November  1980. 

THE  PROBLEM  OF  EXPERIMENTAL  DESIGN  IN  SIMULATION,  CORS/TIMS/ORSA 
Joint  National  Meeting,  Toronto,  May  1981. 

THE  PERFORMANCE  OF  TWO-STAGE  GROUP  SCREENING  IN  FACTOR  SCREENING 
EXPERIMENTS,  American  Statistical  Association  Annual  Meeting, 
Detroit,  August  1981. 

FACTOR  SCREENING  IN  SIMULATION:  PERFORMANCE  OF  A  TWO-STAGE 
RANDOM  BALANCE/PLACKETT-BURMAN  PROCEDURE,  ORSA/TIMS  Joint  National 
Meeting,  Houston,  October  1981. 

AN  EVALUATION  OF  TWO  FACTOR  SCREENING  STRATEGIES  BASED  ON  RANDOM 
BALANCE  SAMPLING,  American  Statistical  Association  Annual 
Meeting,  Toronto,  August  1983. 
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